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(54) Selection facilitation on a graphical interface 

(57) This invention allows the user to rapidly deter- 
mine the interrelatedness of icons on graphical user 
interfaces. Upon selection of an icon, for example in a 
drag-and-drop interface, all other icons to which the 
selected icon may be related are graphically high- 
lighted. This provides a user-friendly, dynamic, visual 
catalog of interrelated icons. A scenario matrix contains 
information that defines the relations among the icons 
on the graphical interface. 
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Description 

FIELD OF THE INVENTION 

This invention relates to computer graphical user s 
interlaces. More specifically, the invention relates to a 
graphical user interface (GUI) allowing users to better 
understand the relationship between icons displayed on 
a computer screen. 

BACKGROUND OF THE INVENTION 

Graphical user interfaces (GUIs) provide ways for 
users of computers and other devices to effectively 
communicate with the computer. In GUIs, available 
applications and data sets are often represented by 
icons consisting of small graphical representations 
which can be selected by a user and moved on the 
screen. The selection of icons often takes the place of 
typing in a command using a keyboard in order to initi- 
ate a program. In general, icons are tiny on-screen sym- 
bols that simplify access to a program, command, or 
data file. Icons are usually activated or selected by mov- 
ing a mouse-controlled cursor onto the icon and press- 
ing a mouse button. 

GUIs include graphical images on computer moni- 
tors and often consist of both icons and windows. (GUIs 
may also reside on the screens of televisions, kiosks, 
and automatic teller machines (ATMs).) A computer 
window is a portion of the graphical image that appears 
on the monitor and is dedicated to some specific pur- 
pose. Windows allow the user to treat the graphical 
images on the computer monitor like a desktop where 
various files can remain open simultaneously. The user 
can control the size, shape, and position of the win- 
dows. 

Although the use of GUIs with icons usually simpli- 
fies a user's interactions with a computer, GUIs are 
often tedious and frustrating to use. Icons must be 
maintained in a logical manner. It is difficult to organize 
windows and icons when many are similarly displayed 
at the same time on a single device. 

In a drag-and-drop GUI, icons are selected and 
moved to a target icon to achieve a desired effect. For 
example, an icon representing a computer file stored on 
disk may be dragged over an icon containing an image 
of a printer in order to print the file, or dragged over an 
icon of a trash can to delete the file. A typical user's 
screen contains many icons, and only a subset of them 
will at any one time be valid, useful targets for a selected 
icon. For example, it would not be useful to drag the icon 
representing a data file on top of an icon whose only 
purpose is to access an unrelated multimedia applica- 
tion. 

OBJECTIVES OF THE INVENTION 

An objective of this invention is an improved system 
and method for organizing and using graphical displays 



on graphical interfaces. 

Another objective of this invention is an improved 
system and method for organizing, displaying, manag- 
ing, and selecting icons and/or windows on a computer 
graphical interface. 

SUMMARY OF THE INVENTION 

This invention permits users to conveniently view 
and manipulate related GUI icons, particularly in a drag- 
and-drop interface but also in any presentation where 
information on the relatedness of icons and/or windows 
is desired. In a preferred embodiment, when one 
icon/window is selected and/or dragged, other related 
icons are then distinguished by one of a plurality distin- 
guishing features, such as by brightening, by rounded 
corners, or by an oval shape and/or highlighted text. 
(Note that in this disclosure, "distinguishing feature" and 
"highlight" are used interchangeably.) This distinguish- 
ing allows the user to see and understand which icons 
are related to the target icon, thus creating an interac- 
tive visual index for the user and/or guiding the user to 
specific targets (drop sites) in drag-and-drop interlace. 
Icons are related when they represent data, functions, 
and/or programs that can be used together. The user 
can also specify which icons are related, e.g., those in a 
similar class like office applications, word processing, or 
drawings. Strength of the relationship can also be indi- 
cated by factors such as icons being most frequently 
used or recently used together. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of the hardware and one 
preferred embodiment of the present invention. 

Figure 2 is a drawing ol two monitors with graphical 
images/selectable items on their graphical interfaces. 

Figure 3 is a flow chart showing the steps of one 
preferred method of the present invention. 

Figure 4 is a block diagram of a data structure that 
defines the interrelatedness of icons. 

Figure 5 is a flow chart showing the steps of one 
preferred method for determining most likely pairs of 
icons to be used. 

DETAILED DESCRIPTION OF THE INVENTION 

This invention permits users to conveniently view, 
organize, use, and understand relationships between 
GUI objects including icons with static or animated 
graphics, text, multimedia presentations, and TV broad- 
casts. GUI objects could also include three dimensional 
images, for example, those used in virtual reality appli- 
cations. When one data object or program, having a dis- 
crete graphical appearance on the GUI. is selected or 
dragged, target icons are highlighted. Generally, but not 
necessarily, the highlighting lasts for a time period lim- 
ited to the time the original data object is 
selected/dragged. Th highlighting is used to make the 
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user aware that a relation exists between the high- 
lighted icons. Further the highlights are used to guide 
the user toward appropriate target icons. 

The present invention is capable of running on any 
general purpose computer system or microprocessor 5 
controlled television, including units which have the abil- 
ity to present multimedia information. One preferred 
embodiment is schematically represented in a block 
diagram in Figure 1 . A computer system 1 00 comprises 
a control processing unit (CPU) 110, memory storage io 
device 120, one or more monitors or graphical inter- 
faces 130, and selection device 140 such as a mouse or 
speech recognition system 1 78. In one embodiment, a 
IBM RISC SYSTEM/5000 100 comprises a control 
processing unit (CPU) 110, memory storage device 75 
120, one or more monitors 130, and a mouse 140. The 
mouse 140 may be used to select graphical images like 
icons 160 or windows 180. (Note that graphical images 
are also referred to as selectable items herein.) On an 
IBM RISC System/6000 multiple monitors 130 can be 20 
controlled by multiple monitor adaptor cards 115 such 
as the IBM RISC System/6000 Color Graphics Display 
Adaptor. The computer system 100 may also have 
audio input/output capability 170. An ActionMedia II Dis- 
play Adapter 175 (described in the IBM ActionMedia II 25 
Technical Reference) can be used for audio/video play- 
back 170. 

This adaptor 175 may also be used to display TV 
broadcasts/signals 190 and other full motion video and 
sound audio/visual on the monitors 130. 30 

In addition, speech recognition 178 may be pro- 
vided (such as the IBM VoiceType Dictation Adapter). 

In an alternative embodiment, the CPU 110 can be 
connected to 112 a client/server network (or LAN 111) 
to which other target monitors 130 and/or systems 100 ss 
are connected. 

Systems 100 that can be used to display graphical 
images, like icons and windows, are well known. 

GUIs can be used to control any apparatus having 
a monitor. In the field of television (TV), channel selec- <o 
tion can be affected by selecting an icon consisting of 
the animated video broadcast on that channel fre- 
quency. 

Figure 2 is a block diagram of a system 200 show- 
ing graphical images/selectable items (such as icons as 
160 or windows 180, both which may contain animated 
images) appearing on a primary monitor 230. A selecta- 
ble icon 160 or window 180 may be selected 204 and 
dragged 205 on the primary monitor 230 using a mouse 
140 or other selecting device 140. so 

The following explains the invention using an icon, 
although the concept applies to other graphical images 
such as windows. Once an icon 160 is selected 204, 
certain icons in the set of displayed icons 261 are high- 
lighted 262 (by color changes or other graphical means) 55 
if they are related to the icon 160 or can receive the 
select icon 160 in a drag and drop icon interface. For 
example, icons 262 may be the only valid "drop sites" for 
selected icon 160. The relation between icons is defined 



as described in Figure 4 below. The set of displayed 
icons 263 may reside on another monitor 240 con- 
nected to the same computer (local) or c nnected over 
a computer network (remote). 

The relation between icons may be defined by a 
user profile 271. In a preferred embodiment, this is a 
"scenario matrix" 270 on the memory storage device 
120. This user profile 270. which can exist as a data file, 
may consist of an array of numbers indicating which 
icons can receive other icons in a drag-and-drop GUI or 
which icons are related to one another in a more gen- 
eral GUI. 

Figure 3 is a flow chart showing the steps per- 
formed by an icon facilitator program 300 executed by 
system 100 or 200. 

The icon facilitator program 300 first queries 310 if 
an icon 160 is selected, for example by a mouse-or 
voice-controlled cursor. Methods for determining if an 
icon is selected are well known. If the icon 160 is 
selected 31 1, the program identifies 320 an icon identi- 
fier, like ah icon index number, and locates 330 the cor- 
responding entry, a strength value, in the scenario 
matrix 270 (or user profile 271) which may be stored as 
a file on medium 1 20. (An icon index number can be any 
name that uniquely identifies the icon.) The searching of 
indices in files containing arrays or records of data is 
known to those skilled in the art. This file may be 
updated periodically by application software or the user. 
(If an icon is added to the GUI, a new entry may be 
added to the matrix 270.) As a result of step 330, related 
icons are then highlighted 340. In. other words, high- 
lighted icons are those with a relationship with the 
selected icon, where the relationship is indicated by the 
strength value corresponding to a pair containing the 
selected icon and the related icon. Highlighting of icons 
may be accomplished using routines supplied by any 
standard window software (See for example, Nye, A. 
(1988) Xiib Programming Manual. O'Reilly & Associ- 
ates: Cebastopol, CA which describes color manipula- 
tion for the purpose of highlighting). If no icons 312 are 
selected, all icons are unhightighted. 

Figure 4 is a block diagram of one typical preferred 
data structure 470 that lists alphanumerical indices 475, 
i 476, for each selectable item 160 and alphanumerical 
indices 480, j 477, for each of zero or more possible tar- 
get icons (261, 263). This matrix of numbers 470 is 
called a a scenario matrix, and is used for highlighting 
related icons when a user selects an icon. The scenario 
matrix may be a file on disk, or data in memory, with 
designators, like numbers, typically 490 and 491, indi- 
cating which icons are related for the purposes of high- 
lighting relevant target icons. 

For example, consider a matrix A(i.j) 470 where i 
476 and j 477 are numerical indices denoting various 
icons. The scenario matrix can be a square and sym- 
metric matrix, i.e. A(i.j) = AG.i) . if all icons i and j are 
related to each other, for example, rf i can be dragged on 
top of j. and j can be dragged on top of i with a useful 
result. More generally the matrix is not symmetric if i is 
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related to application j but the reverse is not so. As an 
example, one may transfer artwork from a drawing pro- 
gram into a word processor application, but one may not 
import a document from a word processor program into 
an artwork program. s 

Figure 4 shows a binary scenario matrix 270, 470 
where highlighting (related icons) is denoted by 1 (490) 
and non-highlighting (unrelated icons) is denoted by 0 
(491). When an icon i 476 is selected 204 or dragged 
205, row i 477 of the scenario matrix may be thought of »o 
as a control vector which determines which other icons 
are highlighted, (1=highlight or 0=not-highlight), There- 
fore, once the i-th icon is selected, the highlighting pro- 
gram 300 examines the i-th row of the matrix. 

Icons represented by numerical indices 480 in j is 
entries 477 in this control vector may be distinguished 
with one or more distinguishing features 262, all of 
which can be accomplished by known graphical meth- 
ods in X-windows and other windows environments by 
one skilled in the art of windows programming. These 20 
distinguishing features could be user selectable. Distin- 
guishing features (highlights) might include icon: bright- 
ness; outlining, font, shading, color, size, shape, and/or 
animation. For example, using shape as a distinguish- 
ing feature, an icon shape control, known in GUI inter 1 25 
face programs, may be employed to chapge the shape 
of the icon from one shape, e.g. a rectangle, to another, 
such as a rounded rectangle or an oval. Using fonts as 
a distinguishing feature, a three-dimensional style font 
might be used for the titles of related icons. Also, related 30 
icons could be shaded (or not shaded) in the same way 
as other related icons (and the selected icon). Shading 
of icons is known in GUI interface programming. 
Another distinguishing feature is to brighten the related 
icons, e.g., appropriate target icons (drop sites) 262. 35 
See routines supplied by standard window software 
(see for example, Nye, A. (1988) Xlib Programming 
Manual. O'Reilly & Associates: Cebastopol, CA which 
describes various ways to implement distinguishing fea- 
tures.) This Xlib Programming Manual is incorporated 40 
by reference in its entirety 

In an alternative preferred embodiment, the sce- 
nario matrix 470 may be used with a range of values in 
the matrix. These values, called strength values, could 
indicate a strength of the relationship between any 45 
given pair of icons. For example, the higher the value, 
the more likely it is that one icon is related to another. A 
threshold value may be set so that only those icon rela- 
tions denoted by a sufficiently high value are high- 
lighted. In other words, the strength value between a so 
pair of graphical images must be above a threshold 
value before the graphical images in the pair are consid- 
ered to be related. The scenario matrix (470, 270) may 
be created manually by a user, or created by application 
software. Additionally, the values, 490 or 491 typically, in 55 
the matrix may be provided automatically by a program 
400 which monitors a user's past usage and increments 
values 490 and 491 in the matrix if a user is "likely" to 
use icons j 477 representing various target applications 



after using icons 476 representing various selected 
applications. Other programs 400 could use different 
criteria to change the strength values in the matrix. 

For example, Figure 5 is flow chart indicating how 
the scenario matrix may evolve as program 400 moni- 
tors usage of icons. When the user selects 510 an icon, 
the program 400 checks 520 to see rf the scenario 
matrix value A(i,j) (490, 491) is greater than a threshold 
value T. If yes 530 the appropriate icons are highlighted 
550 as in 340. If no 540. the icons are not highlighted. 
The index 475 of the currently selected icon i is saved 
560 in a variable called savej. When the user makes 
the next selection 570 the program 400 again checks 
580 to see if the scenario matrix value A(i,j) is greatei 
than a threshold value T. If yes 590 the appropriate 
icons are highlighted 594 as in 340. If no 592, the icons 
are not highlighted. The index 480 of the currently 
selected icon is saved 596 in a variable called savej. In 
step 598, the value A(save_i, savej) of the scenario 
matrix 470 is incremented at location (savej, savej). 
In this manner, the scenario matrix reflects "most fre- 
quently used pairs" and can therefore highlight target 
icons 262 most likely to be used in, for example, a drag 
and drop interlace. Here the strength of the relationship 
between any given pair of icons is determined by the 
number of times the pair of icons is used together. 

Alternatively, the values in the scenario matrix can 
be normalized by the sum of all the array elements. In 
this manner, the array value reflects the number of times 
a pair is selected as a percent of the total number of pair 
selections. 

Using similar methods, "most recently used pairs" 
may also be cataloged and highlighted. This may be 
accomplished, for example, by periodically resetting the 
scenario matrix values 490 and 491 to some default and 
then letting them accumulate again as in 598. Alterna- 
tively, the scenario matrix 470 may be a three-dimen- 
sional array, whose third dimension 478 contains 
values, typically 479, indicating relationships for a 
period of time. For example, a 2-dimensional array, typ- 
ically 485, would exist for each of one or more given 
periods of time. In this preferred embodiment, to deter- 
mine "most recently used pairs", step 330 locates icons 
in the 2-dimensional array 485, corresponding to the 
most recent time period 486 in order to determine which 
icons have the strongest relation {here most recent) and 
therefore are highlighted 340. Selection of other time 
periods, or groups of time periods, can be achieved by 
specifying specific 2-dimensional arrays 485 or time 
period ranges. 

In addition, the data files 270 containing the sce- 
nario matrix 470 can be made portable so that the entire 
scenario matrix can be moved from machine to machine 
or otherwise used to customize a machine's GUI for a 
particular user, or to provide a common "base" scenario 
matrix for a company or organization. The scenario 
matrix would b in the form of a data file or shared mem- 
ory file that can accessed across a LAN, WAN, or other 
network. For example, in the UNIX operating system, it 
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is standard procedure to access remote file systems via 
NFS (network file system) protocols. In particular, 
remote files can be accessible via ioctl() function calls. 

Given this disclosure alternative embodiments 
would become apparent to one skilled in the art. 5 ■ 

Claims 

1 . A computer system comprising: 

10 

a central processing unit and a memory; 

one or more graphical interfaces each having 
two or more graphical images, one of the 
graphical images being a selected graphical is 
image and one or more of the graphical images 
being a set of displayed graphical images, one 
or more of the displayed graphical images 
being a related graphical image having a rela- 
tion with the selected graphical image; 20 

a data structure residing on the computer 
memory that defines how the selected graphi- 
cal image and each of the related graphical 
images are related; and 25 

a process for changing one or more distin- 
guishing features of all the related graphical 
images when the selected graphical image is 
selected. so 

2. A computer system, as in claim 1 , where the rela- 



3. 



4. 



5. 



6. A computer system, as in claim 1 , where the distin- 
guishing feature includes any one of the following: 
brightness, outlining, font, shading, color, size, 
shape, and animation. ss 

7. A computer system, as in claim 1. where the 
selected graphical image is selected by a pointing 
device. 



8. A computer system, as in claim 1, where the 
selected graphical image is selected by a speech 
recognition system. 

9. A computer system, as in claim 1 , where the graph- 
ical interface is a television. 

1 0. A computer system comprising: 

a central processing unit and a memory; 

one or more graphical interfaces each having 
two or more graphical images, one of the 
graphical images being a selected graphical 
image and one or more of the graphical images 
being a set of displayed images, one or more of 
the displayed graphical images being a related 
graphical image having a relation with the 
selected graphical images; 

a data structure, residing on a computer mem- 
ory, having one or more strength values, each 
of the strength values indicating a strength of 
the relationship between each of one or more 
pairs of graphical images, each pair including 
the selected graphical image and one related 
graphical image; and a process for changing 
one or more distinguishing features of each of 
the related graphical images when the selected 
graphical image is selected. 

11. A computer system, as in claim 10, where the each 
of the related graphical images is in a pair with a 
strength value above a threshold. 

12. A computer system, as in claim 10, where one or 
more of the strength values is determined by a 
number of times the graphical images in a pair are 
selected as a pair. 

13. A computer system, as in claim 12, where th 
strength values are determined for a time period. 

14. A computer system, as in claim 13, where the time 
period is a most recent time period. 

15. A method of showing a relationship between icons 
on a graphical interface of a computer system com- 
prising the steps of: 

selecting a selected graphical image that is dis- 
played on the graphical interface; 

determining a relation between the selected 
graphical image and one or more related 
graphical images displayed on the graphical 
interface, the relation defined by a strength 
value in a matrix of strength values, each 
strength value indicating the strength of the 



A computer system, as in claim 1 , where the rela- 
tion is that the related graphical images are drop 
sites for the selected graphical image. 

35 

A computer system, as in claim 1 , where the rela- 
tion is that the related graphical images are in a 
same class as the selected graphical image. 

A computer system, as in claim 1 , where the rela- <o 
tion is that one or more related graphical images 
represent a related program that can be used with a 
selected program represented by the selected 
graphical image. 

as 

A computer system, as in claim 1, where the rela- 
tion is that one or more related graphical images 
represent a related data that can be used with a 
selected program represented by the selected 
graphical image. so 
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relationship between a pair of graphical 
images, each pair including the selected graph- 
ical image and related graphical image; and 

distinguishing the related graphical images by s 
changing a distinguishing characteristic of 
each of the related graphical images when the 
selected graphical image is selected. 

16. A computer system comprising: , 10 

a central processing unit means and a memory 
means; 

one or more graphical interlace means for dis- is 
playing graphical images, each graphical inter- 
face means having two or more graphical 
images, one of the graphical images being a 
selected graphical image and one or more of 
the graphical images being a set of displayed 20 
graphical images, one or more of the displayed 
graphical images being a related graphical 
image having a relation with the selected 
graphical image; 

25 

a data structure means, residing on the compu- 
ter memory means, for defining how the 
selected graphical image and each of the 
related graphical images are related; and 

30 

a process means for changing one or more dis- 
tinguishing features ol the related graphical 
images when the selected graphical image is 
selected. 
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